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IV&V  BENEFITS  AND  POTENTIAL 
PROBLEMS 


BEN  EF  ITS 


§  Early  detection  of  defects  that  might 
otherwise  remain  undetected  until  later  in 
the  lifecycle  (reduced  cost  and  schedule) 

§  Independence  (technical  point  of  view, 
toolset,  process)  allows  identification  of 
defects  that  could  be  overlooked  by 
development  personnel 

§  Ensures  that  the  delivered  software  will 
meet  each  baselined  software 
requirement 

§  Provides  the  Customer  with  increased 
visibility  into  software  status  and  quality 

§  Reduced  maintenance  cost 

§  Can  serve  to  fill  the  many 

communication  holes  that  result  from 
performance-based  acquisition 


PROBLEMS 


§  Overemphasis  on  independence  can 
result  in  non-productive,  adversarial 
relationship  between  the  IV&V 
contractor  and  the  Development 
contractor 

§  IV&V  analyses  can  be  out  of  sync  with 
Developer  internal  reviews  creating  a 
separate  review  and  rework  process 
that  can  impact  the  development 
schedule 

§  Not  consistent  with  acquisition  reform 
efforts 

§  IV&V  efforts  traditionally  not 

integrated  with  process  improvement 
goals 

§  View  of  the  standard  is  often  different 
on  opposite  sides  of  the  wall 


INDEPENDENT  INTEGRATED 
VERIFICATION  and  VALIDATION  (I2V2) 


wI2V2  attempts  to  address  the  potential  problems  of  IV&V 
without  negatively  impacting  its  benefits 

w  Goals  of  I2V2 

n  Reduce/eliminate  the  schedule  impact  of  IV&V 

n  Provide  for  collaboration  between  the  IV&V  Team  and  the  Developer 

n  Integrate  IV&V  directly  into  the  Developer’s  process  and  process 
improvement  program 


n  Close  coupling  of  production  and  inspection,  reduce  defect  leakage 

n  Eliminate  adversarial  relationship  between  the  IV&V  team  and  the 
Developer 

n  Eliminate  hidden  IV&V  financial  conflicts 


INDEPENDENCE  IN  AN 
INTEGRATED  ENVIRONMENT 


Traditional  IV&V 


|2V2 


Traditional  IV&V  places 
ultimate  importance  on 
independence.  Even  a 
perception  that 
independence  is  not 
maintained  is  frowned  on 
and  viewed  as  a  weakness. 


I2V2  takes  a  balanced 
approach  to  independence. 
It  is  willing  to  accept  a 
perceived  reduction  in 
technical  independence  in 
order  to  gain  previously 
discussed  benefits  such  as 
open  information  exchange, 
interdependence,  and 
teamwork. 


Technical  Independence 

Teamwork 

Management  Independence 

Open  Information  Exchange 

Financial  Independence 

Interdependence 

OPEN  COMMUNICATION  AND 
INFORMATION  EXCHANGE 
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THE  INTEGRATED  TEAM  PROCESS 


Requirements  Release  for 
Preliminary  Design 


7 


OTHER  KEY  PROPERTIES  OF  I2V2 


w  Common  goals  and  common  definition  of  success 
(interdependence) 

w  Integration  of  I2V2  effort  into  the  Developer’s 
process 

w  Participation  in  internal  Developer  reviews 

w  Integration  of  I2V2  effort  into  Developer’s 
process  improvement  program 

w  Striving  for  a  common  I2V2  and  Developer 
assessment  of  risks  and  status  to  Customer 


INDEPENDENCE  ^INTERDEPENDENCE 


w  With  Traditional  IV&V  it  is  possible  for  the  IV&V  Team 
to  succeed  while  a  program  fails 

n  The  IV&V  Team  can  succeed  by  identifying  a  large  number  of 
Developer  problems  (“I  told  you  so”) 

n  The  IV&V  Team  has  no  accountability  for  system  failure 

n  This  is  the  basis  for  many  adversarial  relationships  between  IV&V 
Teams  and  Developers 

w  I2V2  is  built  on  an  interdependent  relationship  between  the 
I2V2  Team  and  the  Developer 

n  With  I2V2,  both  the  Developer  and  the  I2V2  Team  will  either  succeed 
or  fail  together  (achieve  jointly  defined  objectives) 

n  If  the  Developer  is  failing,  it  is  up  to  the  I2V2  Team  to  cooperatively 
develop  approaches  for  resolving  the  associated  problems 

n  In  certain  circumstances,  the  I2V2  Team  may  assume  responsibility 
for  jointly  agreed  to  tasks 


INTEGRATION  OF  I2V2  INTO  THE 
DEVELOPMENT  PROCESS 


WHY  INTEGRATE? 

l2V2 

w  Recognition  that  development  is  an  visibility 
iterative  process  that  starts  well  before 
the  release  of  a  draft  document 
(planning) 

w  The  quality  of  a  product  is  often 
determined  by  choices  that  are  made 
long  before  the  first  artifact  is  produced 

w  Take  advantage  of  multiple 

opportunities  for  I2V2  analysis  and 
feedback  prior  to  release  of  draft  product 

w  Supports  a  goal  of  releasing  a  draft 
document  that  represents  a  consensus  of 
the  I2V2  Team  and  the  Developer 

w  Concurrence  on  methods  of  evaluation 
and  the  standards  of  “goodness” 


Software 

Requirements 

Development 


Traditional 

IV&V 

Visibility 
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PARTICIPATION  IN  INTERNAL  REVIEWS 


WHY  PARTICIPATE? 

§  Reduce  schedule  by  eliminating 
a  separate  I2V2  review  and 
rework  process 

§  Earlier  incorporation  of  I2V2  findings^ 
since  they  will  be  dispositioned  as  a 
natural  part  of  the  process  for  internal 
reviews 

§  Supports  a  goal  of  releasing  a  draft 
document  that  represents  a  consensus 
of  the  I2V2  Team  and  the  Developer 

ADDED  BENEFITS 


l2V2  Participation  Is  Backed  Up  By  The 
Results  Of  l2V2  Analyses 


DETAILED  l2V2  ANALYSES* 
§  Requirements  Analysis 
§  Traceability  Analysis 
§  Interface  Analysis 
§  Hazard  Analysis 
§  Risk  Analysis 

§  Independent  Test  Evaluations 
§  Testability  Analysis 

*Examples  Of  Potential  Analyses 


Detailed  l2V2  Analysis  Results 


J 


•  At  times,  Developers  have  too  many  demands  on  their  time  to  adequately  prepare  for 
Reviews/Walkthroughs/lnspections.  They  have  to  support  multiple  IPTs,  Peer  Reviews, 
and  Walkthroughs,  etc.  while  trying  to  find  time  to  perform  their  own  technical  work 

•  l2V2  analyses  serve  to  supplement  peer  review  comments  providing  developers  with  better 
feedback  on  their  incremental  products 

•  l2V2  can  also  serve  as  a  communications  layer  between  program  IPTs 
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PROCESS  IMPROVEMENT 


w  Traditional  IV&V  is  focused  on  the  detection  and 
documentation  of  defects 

w  I2V2  recognizes  that  modem  process  definition  and  control 
techniques  are  critical  to  developing  high  quality  software 

w  I2V2  can  participate  with  a  software  engineering  process 
group  to  provide  root  cause  analysis  for  common  defect 
types 

w  Root  cause  analysis  can  be  used  to  modify  software 
development  processes  to  prevent  defects  from  being 
created  in  the  first  place 
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PROVIDING  THE  CUSTOMER  WITH 
THE  DATA  THEY  NEED  _ 


w  With  traditional  IV&V,  all  information  flowed  through  the  Customer  so 
lack  of  data  was  never  an  issue;  however,  often  the  data  was  not  at  a 
useful  level  and/or  was  inconsistent  (Developer  view  versus  IV&V  view) 

w  I2V2  focuses  on  providing  the  Customer  with  data  they  need  to  manage 
the  program  without  useless  detail 

n  Direct  data  flow  between  the  I2V2  Team  and  Developer  eliminates 
information  overload  for  the  Customer 

n  Detailed  information  generated  during  I2V2  and  development  efforts  is 
abstracted  to  provide  the  Customer  with  status  assessments  for  key  software 
components  and  key  software  functional  areas;  Green,  Yellow,  Red 
Stoplight  charts  provide  an  excellent  mechanism  for  such  assessments 

n  Action  plans  are  provided  to  address  items  with  Yellow  or  Red  assessments 

n  Whenever  possible,  the  status  assessments  are  presented  as  a  consensus 
position  of  the  I2V2  Team  and  the  Developer  eliminating  inconsistent  data 
inputs  to  the  Customer  that  forces  them  to  arbitrate  between  two  conflicting 
views 
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I2V2  -  PUTTING  IT  ALL  TOGETHER 
(PRELIMINARY  DESIGN  EXAMPLE) 


Develop 
Prelim 
Design  Arch 


Proceed 

To 

Detailed 

Design 


Background 

§  Traceability  Analysis 

§  Risk  Analysis 

§  Interface  Analysis 

§  Test  Planning  Analysis 

l2^  Analyses 

§  Hazard  Analysis 

▼ 

Reviewed 

Design 

Artifacts 

Ready  For 

Successful 

PDR 


Independent  Integrated  Verification  and  Validation  (l2V2)  is  the  application 
of  an  IV&V  process  integrated  within  the  software  development  process. 
The  l2V2  team  actively  participates  in  all  aspects  of  the  software 
process  in  order  to  detect  and  resolve  errors  before  they  are 

captured  in  deliverable  products. 
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COMPARING  I2V2  TO  IV&V 
(PRELIMINARY  DESIGN  EXAMPLE) 


l2V2  H 

^ _  Schedule 

Savings 


IV&V 


Review  and  Rework 
Cycle 
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SOME  IMPLICATIONS  OF  I2V2 


w  Rapid  I2V2  analyses 
w  Challenges  and  limitations 
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RAPID  I2V2  ANALYSES 


w  Background  I2V2  analyses  can  provide  incremental 
analysis  results  to  support  quick  response  internal 
reviews 

w  Focus  quick  response  analyses  to  reduce  analysis 
time;  focus  based  on: 

n  Anticipated  problem  areas  based  on  past  experience 
n  Known  problem  areas  based  on  earlier  analyses 
n  Risk  Analyses 

w  Use  tools  to  automate  and  speed  up  analyses 

n  Developer  Tool  Set 
n  V&V  Tool  Set 

n  Microsoft  Tool  Set  (otherwise  known  as  Office) 


CHALLENGES  AND  LIMITATIONS 


w  Must  get  “buy-in”  from  all  parties  (Developer,  Customer,  I2V2  Analysts) 
w  Must  have  trust  and  honor  in  all  parties 

w  Must  clearly  define  roles  and  responsibilities  of  the  I2V2  Team  versus 
that  of  the  Developers  to  avoid  conflicts 

w  Success  has  to  be  defined  as  a  win-win- win  proposition 

w  Strong  interpersonal  skills  are  a  must  for  all  parties 

w  Must  have  highly  skilled  I2V2  analysts 

n  Developer  will  never  enter  into  this  close  relationship  with  technically 
inferior  analysts 

w  I2V2  analysts  may  need  to  co-locate  at  Developer  site 
w  Analysis  can  only  find  so  many  problems 

n  I2V2  analysts  must  realize  that  they  are  products  of  their  past  experience 
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I2V2  EXPERIENCE 


w  I2V2  Teams  successfully  participated  in 
two  recent  Army  software  development 
programs 

w  I2V2  supported  Developer  process 
improvement  initiatives  transitioning  from 
CMM  Level  1  to  Level  4  over  a  seven  year 
period 

w  Met  all  key  schedule  dates  under  program 
control 

w  Successful  IOT&E  and  multi-year 
production  awards 

w  Cooperative  Research  and  Development 
Agreement  (CRD A) 

w  Nominated  to  Crosstalk  as  one  of 
Government’s  top  five  software  projects 

w  I2V2  Team  provided  a  built-in  surge 
capability  for  the  program 


I2V2  -  A  PROPOSED  EXTENSION 


w  IV&V  often  uses  artifact  analysis  to  predict  downstream  problems 
(e.g.,  if  requirement  X  isn’t  traced  to  the  design,  it  won’t  be 
implemented) 

n  Analysis  is  heavily  based  on  past  program  experience 
n  Difficult  to  recruit  qualified,  experienced  staff  to  do  analysis 
n  Ignores  data  or  knowledge  not  in  IV&V  documentation  domain 

w  I2V2  could  be  supplemented  by  a  “shadow”  development  team 
working  1-2  process  phases  ahead  of  main  development 

n  Personnel  from  Developer  and  I2V2  team  members 

n  Verify  the  analytical  “predictions”  correctness 

n  Tune  the  analysis  methods  by  flowing  findings  back  to  main 
development  team 

n  Detect  problems  not  found  by  analysis  (e.g.,  unplanned  test  tools) 
n  Refme/validate  downstream  process  and  standards  (e.g.,  unit  testing) 
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I2V2  -  A  PROPOSED  EXTENSION 

(Continued) 

w  Demonstration/Prototype  Products 

n  Most  programs  rely  heavily  on  these  for  customer  assessment 
n  Often  done  with  totally  different  processes 

n  Having  I2V2  members  on  these  teams  could  greatly  benefit  program 
i  Potentially  allow  for  releasable  products  for  certain  uses 
i  Tune  the  I2V2  analysis  to  focus  on  actual  versus  theoretical  problems 
i  Recruit  and  retain  better  I2V2  staff 

i  Allow  for  more  research  on  cooperative  rapid  prototyping 

w  Early  detection  of  integration  problems 
w  Validation  of  artifact  quality  and  usefulness 
w  Risk  reduction 
w  Faster  maturation  of  software 


21 


THE  INTEGRATED  TEAM  PROCESS 
(WITH  SHADOW  TEAM) _ 


Feedback 


Requirements  Release  for 
Preliminary  Design 
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CONCLUSIONS 


w  IY&V  is  an  effective/proven  methodology 

w  Where  workable,  I2V2  provides  a  refinement  to  the 
IV&V  process  that  addresses  aspects  of  IV&V  that 
have  presented  problems  in  the  past 

w  I2V2  meets  much  of  the  definition  of  independence 
without  the  barrier  to  communication  of  “the  wall” 

w  The  right  team  members  and  managers  are  needed 

w  Offers  an  alternative  that  addresses  declining  use  of 
rigid  standards 


I2V2  -  WHERE  DO  WE  GO  FROM  HERE? 


w  Article  is  currently  in  work  to  provide  a  more 
detailed  discussion  of  the  key  elements  of  I2V2 

w  Written  I2V2  procedures  to  be  generated  in  parallel 
with  application  on  future  programs 

w  Proposing  the  addition  of  I2V2  as  a  named  V&V 
form  in  the  next  update  of  IEEE  STD  1012-1998 
(nothing  in  the  current  version  of  the  standard 
precludes  I2V2) 

w  Future  project  to  develop  a  tutorial  if  there  is 
sufficient  interest  in  the  I2V2  concept 
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